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Introduction 


The  third  term  of  the  grant  has  continued  to  focus  on  identifying  key  differentiating  performance 
factors  in  the  pre-mastery  and  mastery  phases.  Identifying  these  factors  was  primarily  done  by 
evaluating  survey  results  from  the  first  and  second  round  of  data  collection,  and  analyzing 
performance  on  the  six  stations  including  beginning  cross  analysis  between  stations.  The  lab  was 
able  to  compare  performance  for  current  study  participants  at  one  station  to  those  whose  data 
was  analyzed  from  previously  recorded  in  both  2014  and  in  simulation  based  LVH  repairs. 
Station  design  has  been  updated  and  optimized  to  allow  for  the  best  possible  data  collection. 


Summary  for  Statement  of  Work  Progress  (Read  Progress  Detail  in 

each  Report) 

The  following  section  details  each  element  of  the  SoW  as  it  has  been  addressed  through 
our  work  thus  far.  For  review,  the  following  four  objectives  guided  this  work: 

Objective  One:  To  evaluate  mental  rehearsal  as  an  intervention  for  skill  decay  in  the  pre¬ 
mastery  phase. 

Objective  Two:  To  identify  key  differentiating  performance  factors  for  the  pre -mastery  and 
mastery  phases. 

Objective  Three:  To  develop  a  generalizable,  multi-variable,  predictive  model  of  skills  decay. 
Objective  Four:  To  develop  an  efficient  and  effective  set  of  assessment  tools  and  individualized 
training  recommendations  to  counteract  skills  decay. 

In  its  third  year,  our  work  has  continued  to  focus  on  Objectives  Two  and  Four  because 
significant  progress  on  these  two  objectives  is  required  in  order  to  address  Objectives  One  and 
Three.  Begin  granted  a  no  cost  extension  until  09/2016,  we  intend  to  shift  our  primary  towards 
addressing  Objectives  One  and  Three  as  data  analysis  continues  as  this  project  completes. 

Objective  One 

No  progress  at  the  time  of  this  report.  We  plan  to  address  this  objective  as  data  collection 
continues. 

Objective  Two 

The  greatest  area  of  progress  for  Objective  Two  has  been  compiling  the  data  collected  in 
2014  and  2015  along  with  conducting  analyses.  During  data  collection  common  error  checklists 
were  completed,  and  the  checklists  were  later  used  to  code  and  timestamp  key  events  or  errors 
using  video  collected  from  each  clinical  scenario  station.  Videos  from  each  station  were  also 
coded  for  the  major  steps  that  occurred  in  each  participant’s  procedure.  The  coding  of  procedural 
steps  for  each  participant  will  allow  for  comparison  between  performance  level  and  post¬ 
graduate  year.  To  assist  in  further  differentiating  performance  factors  between  the  pre -mastery 
and  mastery  phases,  performance  data  was  collected  from  nine  retired  general  surgeons.  These 
retired  surgeons  represent  complete  mastery  with  varying  levels  of  non-practice,  ranging  from  2 
years  out  of  clinical  practice,  to  23  years  out  of  clinical  practice.  Data  from  retired  surgeons  will 
be  coded  and  analyzed  in  a  fashion  identical  to  that  of  resident  participants.  The  level  of 
performance  decay  can  then  be  compared  between  residents  in  the  pre-mastery  phase  and  retired 
surgeons  who  achieved  complete  mastery  while  in  practice. 
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Methods  of  Analyzing  Data 
Needle  Angle:  Central  Line 

Common  events  and  errors  have  been  logged  in  RedCap  during  the  2015  data  collections 
regarding  the  Central  Line  Catheterization  procedure.  Time  points  still  need  to  be  incorporated. 
When  this  is  completed,  analysis  on  performance  between  2014  and  2015  participants  can  be 
performed. 

One  area  of  variation  between  participants  was  the  needle  angle  for  accessing  the  vein  in  the 
central  line.  Research  assistants  analyzed  the  needle  angle  and  number  of  attempts  to  access  the 
vein.  Time  segments  were  noted  for  further  analysis  of  motor  movements  that  might 
differentiate  those  who  were  successful  from  those  who  were  not. 

Because  pauses  can  signify  surgical  planning,  research  assistants  viewed  all  the  videos  of  the 
central  line  to  identify  those  time  points  in  which  the  participant  was  pausing  while  viewing 
either  the  instruments  or  the  simulator  and  not  interacting  with  the  experimenter  so  that  further 
motion  analysis  can  be  performed  to  assess  the  length  of  time  taken  during  each  pause. 

Participants  varied  in  the  technique  that  they  used  to  access  the  vein.  Analysis  of  videos  of 
needle  insertion  evaluated  the  trajectory  and  perseveration  of  tissue  disturbance. 

Participants  varied  in  ability  to  trouble  shoot  central  lines.  Analysis  of  technical  error  rate  and 
ability  to  troubleshoot  central  line  insertion  evaluated  for  any  associated  correlation. 

A  regression  of  the  errors  committed  while  accessing  the  central  line  and  the  motion  smoothness 
as  measured  by  motion  monitoring  sensors  measures  the  association  of  categorical  and 
continuous  measures  of  surgical  performance. 

A  survey  on  critical  steps  and  events  for  the  Central  Line  Catheterization  procedure  was 
developed  to  create  a  procedure  score.  Multiple  procedural  experts  have  completed  the  survey 
and  it  is  currently  in  the  validation  phase. 


Bowel  Anastomosis 

Common  events  and  errors  have  been  logged  in  RedCap  during  the  2015  data  collections 
regarding  the  Bowel  Anastomosis  procedure.  Statistical  analysis  on  performance  differences  and 
similarities  for  2014  and  2015  participants  is  starting. 

Bowel  Anastomosis  can  be  performed  in  a  couple  of  valid  ways.  The  creation  of  decision  trees 
for  each  of  these  methods  helps  the  categorization  of  surgical  decision  making. 

Evaluation  of  critical  decisions  such  as  suture  type  assesses  surgical  knowledge.  Video  and  final 
product  analysis  judged  whether  participants  sutured  one  or  two  layers  and  whether  absorbable 
or  non-absorbable  sutures  were  used  for  the  different  layers. 
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A  survey  on  critical  steps  and  events  for  the  Bowel  Anastomosis  procedure  was  developed  in 
order  to  create  a  procedure  score.  Multiple  procedural  experts  have  completed  the  survey  and  it 
is  currently  in  the  validation  phase. 


Urinary  Catheter 

Common  events  and  errors  have  been  logged  in  RedCap  during  the  2015  data  collections 
regarding  the  Urinary  Catheterization  procedure.  Statistical  analysis  on  performance  differences 
and  similarities  for  2014  and  2015  participants  is  starting. 

Participant’s  medical  decision  trees  regarding  urinary  catheter  placement  in  hypothetical 
scenarios  were  transcribed  and  categorized.  Comparison  to  standard  treatment  and  possible  risks 
will  identify  gaps  in  knowledge. 

Knowing  when  to  refer  to  a  specialist  is  a  critical  decision  in  medicine.  Problem  solving  in  the 
case  of  common  and  unique  hypothetical  scenarios  involving  urinary  catheterization  was 
analyzed  for  medical  decision  tree  and  the  tolerance  of  ambiguity  before  making  a  referral. 

A  survey  on  critical  steps  and  events  for  the  Urinary  Catheterization  procedure  was  developed  in 
order  to  create  a  procedure  score.  Multiple  procedural  experts  have  completed  the  survey  and  it 
is  currently  in  the  validation  phase. 


LVH  Repairs 

Common  events  and  errors  have  been  logged  in  RedCap  during  the  2015  data  collections 
regarding  the  Laparoscopic  Ventral  Hernia  repair  procedure.  Statistical  analysis  on  performance 
differences  and  similarities  for  2014  and  2015  participants  is  starting. 

Identification  of  movement  gestures  in  motion  tracked  data  in  laparoscopic  simulator:  We 
developed  analysis  tools  to  automatically  identify  segments  of  goal  directed  movements  during 
clinical  scenario  tasks.  The  method  employs  a  combination  of  measures,  including,  speed, 
frequency,  and  the  spatial  extent  of  movement,  in  order  to  determine  time  points  of  the  total  data 
record  that  are  likely  associated  with  movement  initiation,  ballistic  action,  and  stabilization  of 
movement. 

For  all  participants  that  attempted  the  laparoscopic  ventral  hernia  repair  (LVH)  from  2014, 
bimanual  and  unimanual  coordination  during  the  mesh  securing  phase  have  been  identified  using 
the  Transana  software.  Workers  observed  video  collected  from  each  participant  and  identified 
time  points  when  modality  changed  and  are  compiling  the  time  points  into  a  spreadsheet  for 
future  motion  analysis. 

Analysis  of  angle  and  rotation  of  hands  during  use  of  surgical  instruments  to  grasp  suture 
associate  the  minimum  amount  of  rotation  required  to  efficiently  grasp  suture  in  the  course  of 
repairing  a  hernia  on  a  simulated  laparoscopic  ventral  hernia  repair. 
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A  survey  on  critical  steps  and  events  for  the  Laparoscopic  Ventral  Hernia  repair  procedure  was 
developed  in  order  to  create  a  procedure  score.  Multiple  procedural  experts  have  completed  the 
survey.  Following  one  more  expert  survey  completion,  validation  will  begin. 

Laparoscopic  Ventral  Hernia  repair  skins  for  2015  participants  were  graded  using  two  raters. 
Inter-rater  reliability  identified  two  categories  that  required  additional  grading  and  improvement. 
New  kappa  scores  reflect  statistically  sound  grading.  The  next  step  is  to  compare  repair  scores 
between  2015  and  2014  participants. 


Matlab 

In  order  to  utilize  the  motion  monitoring  data,  Matlab  code  previously  developed  to  capture 
motion  metrics  (e.g.,  path  length,  working  volume,  idle  time)  for  another  project  is  currently 
being  adapted  for  its  use  with  each  station  on  the  DoD  project.  When  the  Matlab  code  is  finally 
up  and  running,  time  points  for  each  station  collected  during  previous  quarters  will  then  be 
utilized  to  identify  valuable  segments  for  motion  analysis  and  the  code  will  produce  meaningful 
information  regarding  path  length,  working  volume,  and  idle  time  for  each  segment. 


Virtual  Reality  Station 

The  2014  data  was  analyzed  to  set  up  a  performance  review  tool  for  the  force  tasks.  The  just 
noticeable  difference  task  data  was  analyzed  to  give  every  participant  a  perspective  of  the 
smallest  difference  in  stiffness  they  could  differentiate.  Based  on  the  results  the  tests  were  altered 
to  have  conditions  with  finer  difference  in  stiffness  for  the  2015  data  collection.  The  force 
production  with  stationary  target  and  the  reaction  time  task  was  analyzed  to  compare  the  path 
lengths  and  distance  from  target  of  each  participant  with  respect  to  given  trial  conditions.  The 
force  production  with  moving  target  module  is  being  analyzed  for  the  3D  path  lengths  and 
accelerations. 

Previously  developed  human  error  classification  systems  are  currently  being  used  to  deconstruct 
errors  at  major  decision  points  in  the  simulated  laparoscopic  ventral  hernia  repair  (LVH).  The 
taxonomy  (see  Table  1)  distinguishes  between  cognitive  and  technical  failures  that  occurred 
during  each  participant’s  performance.  Using  Transana,  time  points  will  be  collected  which  will 
later  parse  the  large  procedure  video  into  more  manageable  video  clips  that  show  useful 
highlights  of  each  error  characteristic. 
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Table  1:  Error  framework  for  classifying  error  type  and  error  level 


Error  characteristic 

Definition 

Error  type 

Omission 

Commission 

Failure  to  perform  a  step  entirely 

Failure  to  perform  a  step  correctly 

Error  level 

Cognitive 

Information 

Inability  to  detect  cues  arising  from  change  in 

Diagnosis 

system  state 

Inaccurate  diagnosis  of  the  system  state  on  the  basis 
of  the  information  available 

Technical 

Strategy 

Selection  of  a  strategy  that  fails  to  achieve  the 
intended  goal 

Procedural 

Execution  of  a  procedure  inconsistent  with  the 

Action 

Mechanical 

strategy  selected 

Failure  to  properly  execute  the  procedure 

Occurrence  of  structural  or  mechanical  failures  that 
do  not  provide  an  opportunity  for  intervention 

We  continued  our  analysis  using  MANOVA  to  determine  which  metrics  were  most  important  in 
positively  identifying  the  demographic  background  of  each  experiment  participant.  We  recently 
enhanced  this  work  with  a  stepwise  procedure  of  including  contributions  from  each  metric, 
depending  on  the  probability  of  type-II  error  for  that  individual  metric.  We  also  employed  a 
covariance  analysis  to  determine  which  metrics  provided  redundant  information. 

An  generalized  estimating  equation  analysis  of  direction  and  amount  of  displacement  during  a 
virtual  reality  manual  compensation  task  shows  that  different  levels  of  force  disrupt  manual 
compensation  more  when  the  force  is  taken  away  than  when  it  is  applied. 


Sensitivity  Analysis 

Individual  differences  in  sensitivity  to  task  factors:  We  performed  a  sensitivity  analyses  of  how 
performance  depended  on  changes  in  task  conditions.  We  found  that  surgery  residents  were 
sensitive  to  the  starting  location  of  movement,  heading  angle  of  movement,  the  degree  of  visual 
distortion,  and  coping  with  changes  in  the  force  threshold  of  the  puncture  task.  Interestingly,  the 
changes  in  movement  heading  and  starting  location  were  significant  factors  across  different 
tasks. 


Transcription 

Three  stations  have  been  transcribed  using  audio/video  transcription  software  Transana. 
Quantitative  and  qualitative  features  from  audio  and  video  recorded  data  from  2014  data 
collections  have  been  coded  and  categorized. 
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Transcription  of  the  focus  groups  that  were  conducted  in  early  2015  has  been  completed  and 
analyzed  to  better  understand  our  recruitment  efforts  and  enhancements  that  are  necessary  for 
our  Performance  Review  Tool. 


Moonlighting 

An  analysis  was  performed  on  the  effect  of  moonlighting  or  on-call  work  in  addition  to  lab  time 
on  skill  decay  in  the  performance  of  the  LVH.  While  the  model  was  not  significant  at  baseline, 
this  analysis  suggests  that  with  similar  data  skill  decay  could  be  seen  as  decrease  in  the  final 
products  (the  LVH  skin)  score  of  the  LVH  by  as  much  as  1 .4  points  per  year.  This  analysis  also 
demonstrated  that  post  simulation  confidence  and  difficulty  ratings  were  far  more  indicative  of 
performance  as  judged  by  the  final  product  than  pre  simulation  ratings. 

Objective  three 

No  progress  at  the  time  of  this  report.  We  plan  to  address  this  objective  as  data  collection 
and  analysis  continues. 

Objective  Four 

Data  Collection  Efforts 

The  primary  focus  of  the  study  team’s  efforts  surrounded  acting  on  lessons  learned  from 
previous  pilot  studies,  training  lab  personnel,  and  streamlining  current  study  protocols  all  in  an 
effort  to  maximize  the  quality  of  the  data  collected  from  our  participants.  Most  notably,  this 
process  included  three  distinct  features: 

1)  Redesigning  the  clinical  procedure  stations; 

2)  Improving  existing  study  protocols/instructions;  and 

3)  Providing  standardized  training  and  assessment  for  all  lab  personnel  who  will  be 
involved  in  data  collection. 


Redesigning  the  Clinical  Procedure  Stations 

Based  on  feedback  from  previous  pilots,  expert  reviews  of  each  station,  and  internal  evaluation, 
each  of  the  clinical  procedure  stations  underwent  a  significant  improvement  over  the  course  of 
quarter  1. 

First,  noted  “experts”  on  each  of  the  four  clinical  procedure  stations  (Central  Line,  Bowel 
Anastomosis,  Urinary  Catheterization,  and  Laparoscopic  Ventral  Hernia)  were  asked  to  provide 
their  feedback  and  suggestions  for  how  each  station  may  be  improved.  For  each  review,  the 
expert  was  asked  a  series  of  semi-formal  interview  questions  by  project  management  both  before 
and  after  performing  the  procedure  (Figures  below  for  examples).  The  audio  and  video  from 
these  expert  reviews  were  transcribed  and  coded  to  generate  a  list  of  possible  improvements  or 
changes  to  the  four  stations. 
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The  second  major  improvement  to  the  clinical  procedure  stations  was  consolidating  and 
simplifying  the  setup  of  each  station.  The  following  examples  show  some  of  the  improvements 
made  to  each  station. 
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Station  A:  Subclavian  Central  Line. 


The  general  setup  for  this  station  remained  the  same;  however,  a  new  task  was  added  to  the  time  rotation  for  this  station’s  effort.  An 
additional  camera  was  affixed  to  a  post  mounted  to  this  station’s  table  to  capture  a  close-up  view  of  the  participant’s  placement  of  needles 
on  the  simulation  model.  This  assists  greatly  in  the  post-processing  video  analysis  of  correct  needle  placement. 


Station  Narrative 

The  introduction  narrative  that  is  read  aloud  by  researchers  at  Station  A  was  shortened.  This  was  done  because  participants  were  getting 
confused  from  all  of  the  information  presented  to  them  in  the  beginning  of  the  station.  Information  that  was  not  considered  pertinent  was 
eliminated  from  the  narrative  in  order  to  keep  residents  present  in  the  situation. 

_ fiiAiiON  -A-  tvinmin  no>  NAWtArivt _  _ ^tTATION  ".A"  biTROlKCnON  N.miLATT\T _ 

Hi,  I'm  (lumek  welcome  to  Station  A.  Here  yoa  wiU  be  ailced  to  place  a  subclav-ian  centtal  line. 

Tbu  model  w  a  retUftic  anaiomtcal  rrpreaentatton  of  a  n|ibt  upper  torao  nnth  uitema]  jugular, 
external  jugular.  aiMl  nibcUvua  veina,  carotid  and  wbeUvian  artertea,  vvnoua  blood  la 
tepreacnlcd  by  Uue  liquid  and  aileilal  blood  by  red  liquid.  Tbe  akin  may  be  pundured; 
bemTvet.  tbe  akm  camaot  be  injected,  marked  or  cut  All  od  ibeae  nutenala  can  be  uacd  during 
ibe  rimubtioQ.  Tbe  aca4wl  does  not  hme  a  blade,  bnt  it  should  be  used  to  go  through  tbe 
motions.  Abo.  the  catheter  can  only  be  inserted  to  the  \evt\  of  tbe  bine  tape.  FinaU>*.  if  there  is 
sometbing  that  you  cannot  do  gi%en  the  timularioo  but  you  would  want  to  do.  thenpleaae 
vethaliae  tbe  action  I  cannot  pnn'ide  anv  feedback  or  atts«*er  questiooa  related  to  tbe  proceduie 
itself.  Do  >oo  have  any  qucftioM  about  what  is  expected  of  you  befose  we  begtn'^ 
tif  Ok.  please  place  your  hands  on  the  sign  while  we  begin  lecotding. 

Once  mseoncher  2  gives  thwnhs  up  to  resoarcher  i 

Otcat.  bete  b  >xMir  clinical  scenario,  von  may  Wsts 


Figure  2:  Old  (Left)  and  New  (Right)  Central  Line  Placement  (Station  A)  Introduction  Narrative 


Suturing  Task 


An  additional  task  was  added  to  the  clinical  procedural  scenarios.  This  task  is  an  isolated  and  constrained  task  of  placing  3  interrupted 
sutures  into  body  tissue  mimics  using  an  instrument  tying  technique.  This  work  is  based  off  of  a  task  developed  and  refined  in  Anne-Lise 
D’Angelo’s  NIH  F32  training  grant  project  using  the  Variable  Tissue  Simulator.  The  task  simulates  differing  levels  of  task  complexity  by 
presenting  two  different  materials  to  suture:  rubber  balloon  (dense  connective  tissue),  tissue  paper  (friable  tissue).  The  addition  of  this 
station  is  highly  useful  in  establishing  a  baseline  of  motor  performance  in  suturing  while  in  an  isolated  contextual  environment.  This  can 
then  be  used  as  a  comparison  to  identify  differences  in  performance  when  suturing  is  investigated  in  the  performance  of  the  Bowel 
Anastomosis  station.  Another  benefit  offered  by  this  station  comes  from  the  data  analysis  methods  refined  for  D’Angelo’s  experimental 
results.  The  suture  board  presented  here  is  identical  to  that  of  her  project,  so  the  analysis  of  the  motion  capture  data  matches  very  well. 
The  analysis  programs  written  for  her  project  can  be  directly  applied  to  data  collected  with  this  additional  station  task. 
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The  performance  of  this  Suturing  Task  was  added  to  Station  A  because  the  Central  Line  placement  task  performance  is  typically  fast  for  a 
participant.  The  spare  time  compared  to  other  stations  was  a  good  place  to  add  a  second  task,  since  participants  are  already  wired  into  the 
motion  capture  system.  The  Suturing  Task  fixture  was  fabricated  to  be  placed  directly  on  top  of  the  Central  Line  fixture,  facilitating  rapid 
change-overs.  The  workspace  provides  participants  with  a  suture  board  containing  materials  and  trays  that  hold  necessary  instruments. 
Additionally,  a  camera  mounted  to  a  support  post  captures  a  close-up  view  of  the  participant’s  suturing  actions. 


Figure  4:  Novice  researchers  instructing  a  2015  participant  on  completing  the  suture  task 
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Station  B:  Bowel  Anastomosis 


An  additional  camera  was  placed  into  this  station’s  workspace  to  capture  close-up  views  of  the  participant’s  hand  movements  as  they 
perform  the  anastomosis  task.  This  helps  to  provide  greater  detail  of  the  bowel  exploration  and  specific  suturing  methods  participants 
choose  to  use  during  the  task  execution.  The  camera  is  affixed  to  a  post  mounted  to  the  station’s  table.  This  is  located  near  the  right 
shoulder  of  the  experimenter  assistant  that  interacts  with  the  subject. 


Figure  5:  New  Bowel  Anastomosis  Simulation  (Station  B)  setup 
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Originally  the  station  involved  residents  using  clips  to  hold  the  bowel  as  they  performed  resections  and  anastomoses.  For  the  purposes  of 
understanding  resident  performance  more  through  guiding  and  instructing  a  researcher,  an  assistant  researcher  was  placed  at  the  Bowel 
Anastamosis  station.  By  introducing  the  role  of  an  assistant  to  the  resident  at  this  station,  the  resident  could  interact  with  them  and  provide 
more  of  their  knowledge  verbally  through  direction. 


Figure  6:  Old  (Left)  and  New  (Right)  Procedure  for  Bowel  Anastomosis  (Station  B) 

In  order  to  improve  post-performance  analysis  of  bowel  repairs,  the  station’s  common  events  checklist  was  revamped  to  consider  the 
different  choices  residents  made,  including  planned  layers,  suture  type,  stitch  type,  completeness,  and  general  procedure  steps.  This  level 
of  detail  will  also  expedite  statistical  analyses  performed  within  the  station  and  across  stations. 


ST.mON  B  COiniON  E^T^TS  CHECKLIST 


DnoioiuiFareaiipaniCfiaitiiuikpqaiyotdieiBiis  iiSocmdiini£iepfO(sdi]R(l&ukpn<>nivsl  aad2b(iaT~UBieiiijiity'ifdiepxtic^>ait 


Das; 


Station  B  Common  Events  Cheddist 


Figure  7:  Old  (Left)  and  New  (Right)  Bowel  Anastomosis  (Station  B)  Common  Events  Checklists 
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Station  C:  Urinary  Catheter 

Based  on  feedback  obtained  from  focus  group  interviews  with  past  participants,  this  station  underwent  a 
major  redesign  in  model  positioning  and  equipment  placement.  First,  the  orientation  of  the  pelvic  model 
was  placed  on  the  table  in  a  fashion  to  mimic  the  positioning  of  a  patient  lying  on  a  bed.  This  better 
simulated  the  leaned  over  position  the  participant  would  have  to  do  if  in  a  patient’s  room.  Second,  a  new 
catheter  kit  was  made  to  better  match  the  way  a  clinician  would  place  a  Foley  kit  between  a  patient’s 
legs  prior  to  starting  a  urinary  catheterization.  The  kit  approximates  the  purchased  kit  and  also  facilitates 
rapid  changeovers  because  it  can  be  picked  up  and  switched  out  with  matching  kits  that  contain  different 
size  catheters  that  can  be  requested.  Additionally,  a  close-up  camera  and  mount  was  added  to  this 
station  to  better  capture  the  detail  in  the  participant’s  catheter  insertion. 


Figures  8-9:  Station  C:  Urinary  Catheter 


Before 


After 
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Addition  of  Close  up  Cameras  to  three  of  the  four  clinical  stations.  These  allow  for  a  more  detailed, 
focused  view  of  the  scene  and  assist  in  coding  errors  and  events.  They  each  have  a  standardized 
position/view  and  are  manually  focused  so  that  participant  movement  does  not  blur  the  view  of  the 
simulation. 


Figure  1 0:  New  close  up  cameras 


Additional  Updates  to  Urinary  Catheter  Station: 

a.  Simulation  models  turned  sideways  (no  longer  facing  edge  of  table)  to  mimic  position  of  actual  patient 
awaiting  catheter  insertion. 

b.  Two  models  used  for  catheterization  instead  of  three  due  to  time  constraints. 

c.  Vaginal  ring  added  to  female  simulator  to  restrict  access  to  urethra;  previously  sutures  were  used  to 
restric  access,  however  these  were  unrealistic  and  prone  to  breaking. 

d.  Yellow  and  red  colored  water  instead  of  clear  water  used  to  represent  normal  and  bloody  urine  return, 
respectively. 

e.  Drainage  bags  removed  from  ports  to  increase  participant  turn-over  and  facilitate  station  clean-up. 

f.  Catheter  boxes/kits  created  to  replace  catheter  stand. 


Figure  11:  Example  of  new  catheter  box 
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In  order  to  better  assess  resident  decision  making,  this  station  underwent  a  redesign  in  equipment  placement.  In 
2014,  residents  were  able  to  see  all  catheter  options  available  to  them.  This  year  residents  were  required  to 
vocalize  their  catheter  choice  without  visualization  of  the  catheters  ahead  of  time.  Only  if  residents  were  unable 
to  articulate  their  decision  did  we  provide  a  list  of  catheter  options. 


Figure  12:  Old  (Left)  and  New  (Right)  Equipment  Layout  for  Urinary  Catheterization  (Station  C) 

Based  on  experiences  researchers  have  had  with  residents  in  previous  data  collections  on  Station  C,  a 
Frequently  Asked  Questions  document  was  created  to  include  questions  that  participants  frequently  ask  about 
the  models  and  urine  output.  By  making  this  document,  researchers  would  have  standard  response  options  when 
prompted  with  these  frequent  questions. 


STATION  “C”  FAQ  RESPONSES 


In  regards  to  questions  about  the  color  of  the  urine; 

“Please  assume  the  color  of  the  urine  is  intentional  and  is  a  part  of  the  simulation.” 


If  the  participant  asks  to  remove  the  ring  on  model  2 ; 

“It  will  be  recorded  that  you  have  asked  to  remove  the  ring,  how'ever  please  proceed  with  the 
ring  in  place.” 

If  this  continues  to  be  a  problem  for  the  participant; 

“Unfortunately  our  simulators  cannot  be  modified,  w'e  will  record  that  you’d  like  the  ring 
removed,  but  please  proceed  as  if  the  ring  were  non-remo\  able.” 


If  there  is  a  potential  problem  with  the  simulator  Ci.e.  ajr  bubble  in  the  bladder); 

“We  imderstand  that  simulators  can  at  times  have  issues.  Please  try  to  re-insert  the  catheter,  or 
re-adjust  its  placement.” 

If  this  continues  to  be  a  problem  for  the  participant; 

“We  apologize  for  any  issues  with  the  simulator;  it  will  be  noted  in  yoiur  records.  If  you  are 
unable  to  proceed  due  to  these  issues,  we  will  continue  onto  the  next  scenario,  station.” 


Figure  13:  Frequently  Asked  Questions  and  Responses  for  Urinary  Catheterization  (Station  C) 
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Station  D:  Laparoscopic  Ventral  Hernia  (LVH) 

Updated  LVH  Pretest  Instructions 

To  gather  more  information  and  clear  information  on  hernia  port  placement,  the  LVH  pretest  instructions  were 
updated.  The  pretest  now  differentiates  between  ports  for  instruments  and  ports  for  cameras.  It  also  asks 
participants  to  assign  the  order  in  which  they  would  place  the  ports  used. 


Directions:  Illustrate  where  you  would  place  your(#)  ports  b 
and  “0”  =  12mni  por^s) 

ased  on  the  location  of  the  hernia  using  '‘X"  =  5mm  portfs) 

IsdsioBBl 

1 

1  v  ^ 

i>l>( 

'  i  ®  j 

1  V  . ' 

Images:  http;//www.hemi  as  urgervscotland.conn/ 


2014 


Directions:  In  the  followirrg  figures,  illustrate  where  you  would  place  your  ports  for  the  repair  portion  of  the  procedure 
based  on  the  locationof  the  hernia  using  the  following  key: 

A  5mm  port  for  instruments  O  12mm  port  for  instruments 

Asmm  port  for  camera  ^  12mm  port  for  camera 

Please  indicate  the  order  in  which  you  would  place  the  ports  by  assigning  a  number  next  to  each  port  placed. 


Epigastric  Hernia 


I  V  . 


Incisioml  Hernia 


I  I 


Umbilical  Hernia 


2015 


Figure  14: 2014  and  2015  LVH  pretests  showing  the  greater  detail  of  the  instructions 
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Improving  Existing  Study  Protocols/Instructions 

The  second  area  where  the  research  team  focused  their  efforts  was  streamlining  and  improving  the  protocols 
and  instructions  for  how  data  were  to  be  collected  at  each  station.  These  improvements  included  creating 
dedicated  Station  Manager  Binders,  establishing  dedicated  researcher  responsibilities  for  each  station,  and 
finalizing  the  Participant  Survey  Workbook. 


Modified  Protocol  to  Standardize  POV  Cameras 

Video  glasses  with  low  image  quality  were  retired  and  replaced  by  additional  head-mounted  cameras.  These 
cameras  were  set  up  to  have  a  better  focus  and  better  image  quality  than  previously  recorded  point-of-view 
video  data. 


Figure  15:  Old  POV  glasses  from  2014  and  new  POV  headbands  for  2015 
Documentation  of  Technological  Protocols 

With  the  addition  of  new  equipment  such  as  the  close-up  cameras,  and  standardization  of  protocols  with  head 
cameras  and  camera  mounting,  a  number  of  new  training  documents  were  created.  These  cover  in  detail  the 
physical  setup  of  technology  devices,  placement  in  the  operating  workspace,  software  configurations,  and  use 
during  data  collections.  Each  of  the  station  personnel  responsible  for  setup  has  been  trained  in  equipment  using 
these  instructions.  Highlighted  below  are  portions  of  documents  concerning  head  and  close-up  camera  setup. 


Figure  16:  Examples  of  documentation  for  technology  setup  used  in  study 
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Lucas-Kanade  Template  Matching  Tracking  Algorithm 

Because  we  have  a  significant  amount  of  data  in  video  format,  we  have  started  using  the  Lucas-Kanade 
algorithm  to  automate  the  identification  of  instruments.  It  is  based  on  template  matching  between  two 
consecutive  frames  and  uses  multiple  parameters  to  identify  transformation  of  feature  frames.  We  were  able  to 
test  this  algorithm  on  Station  A  with  the  scalpel  used  to  perform  the  central  line  catheterization  (Figure  17). 


Figure  17:  Example  of  instrument  tracking  at  Station  A.  The  scalpel  has  been  identified  by  the  algorithm  by  the  blue  rectangular  box 

and  green  dot 

Providing  Training  and  Assessment  for  All  Lab  Personnel  Who  will  be  Involved  in  Data  Collection 

The  third  area  of  work  on  which  the  team  focused  was  in  providing  organized  training  for  any  of  the  lab 
personnel  who  will  be  involved  with  collecting  data.  The  individuals  who  had  been  responsible  for  managing 
each  of  the  four  clinical  procedure  stations  at  the  previous  two  pilots  were  selected  as  “Station  Team  Leaders.” 
Together  with  the  project  and  lab  managers,  these  team  leaders  developed  a  set  of  structured  learning  objectives 
and  training  activities  to  prepare  other  lab  personnel  (i.e.  “Station  Team  Members”)  to  fill  the  role  as  primary  or 
secondary  researcher  for  the  stations. 

New  student  personnel  engaged  in  two  structured  training  periods:  1)  initial  training,  and  2)  follow-up  review 
and  discussion.  Initial  training  was  conducted  by  research  specialists  in  the  lab  in  a  setting  where  each 
individual  received  one-on-one  training  for  their  designated  station.  Follow-up  review  and  discussion  was 
conducted  during  retired  surgeon  data  collections  throughout  the  spring,  where  new  researchers  and  students 
were  able  to  practice  their  roles.  Additional  training  was  provided  when  necessary.  All  in  all,  six  students  were 
trained  to  serve  as  primary  and/or  secondary  researchers  for  each  of  the  clinical  scenario  and  virtual  reality 
stations. 

Continued  training  and  cross  training  of  new  personnel  into  station  teams  will  occur  throughout  the  months  of 
April  and  May. 
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Continued  updating  “Station  Manager  Binders” 


To  reflect  best  practices  including  more  detailed  inventory  and  organization  to  the  mobile  units,  the  station 
manager  binders  were  continually  updated.  New  protocols  were  documented  for  how  to  set  up  and  run  each  of 
the  four  clinical  stations.  Pre-  and  post-  collection  checklists  were  also  added  to  ensure  thoroughness  and 
organization.  Binder  contents  were  specialized  and  further  detailed  for  each  specific  station. 


Contents  of  This  Binder 


TabNi-mbe« 

CoKirwis 

One 

Seti^)  PovferPoint 

Two 

SUtkm  Manner  Notes  for  Data  CoOectioin 

Thic* 

.additional  Training  Materials  (e.g.  Motion  Monitor) 

Foot 

Cart  Ins'entoo' list 

2014 


BINDER  CONTENTS 

Station  A  -  Central  Line 


Tab 

Couteuts 

Pre-Collection 

Preparation  Checllist 

Cart  Imentorj'  Checklist 

Suture  Material  Preparation 

Cart  Organization 

Station  Setup 

Station  Setup  Diagrams 

Filling  the  Simulator:  Instructions 

Motion  Monitor  Setiqi 

Camera  Frame  and  Webcam  Seti^ 

Data  Collection 

Station  A  Responsibihties  (Data  CoDecdoii 
Protocol) 

Cognitive  Scenarios  by  Data  Collection  Site 
Sasing  Motion  Monitor  Files:  Instructions 
Common  Events  Checkhsts 

Suture  Board  Preparation  and  Placonent 
Photographing  arid  Stormg  Sutured  Materials 
Station  Manager  Notes 

Post-C  ollection  Resources 

Post-Collection  Checklist 

Stabnn  Signage 

2015 


Figure  18: 2014  and  2015  binder  contents  page  describing  in  greater  detail  what  all  station  training  binders  contain 


Resident  Participant  Workbook  Addition 


The  resident  workbook  was  supplemented  this  year  with  questions  regarding  the  number  of  procedures 
performed  and  observed  by  each  resident.  This  allows  for  a  clearer  understanding  of  the  experience  level  each 
resident  has  and  what  previous  experience  they  have  with  each  station.  Additionally,  the  residents  were  asked  to 
rate  their  confidence  in  different  suturing  techniques  and  with  different  tissue  types. 


Pl*««  •sllmalt  th«  numb«r  of  tlm€s  that  you  hav*  ptrformeil  and  Sub)act  ID 

observed  the  following  proctduras: 


Figure  1 9:  Workbook  Addition  to  Resident  Participant  Workbooks 
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Retired  Surgeon  Workbooks 


The  retired  surgeon  workbooks  were  developed  based  on  the  resident  participant  workbooks;  however,  the 
survey  questions  were  modified  to  match  a  retired  surgeon’s  experience  and  lifestyle.  The  goal  was  to  keep  as 
much  information  in  the  retired  surgeon  workbooks  the  same  as  the  resident  participant  ones  so  that  analyses 
could  be  performed  across  groups.  The  workbook  and  its  modifications  were  submitted  and  approved  by  the 
University  of  Wisconsin  Institutional  Review  Board. 


University  of  Wisconsin 

DEPARTMENT 
OF  SURGERY 


J  □ 


Figure  20.  Development  of  Retired  Surgeon  Workbook  (Top)  Modeled  After  Resident  Participant  Workbook  (Bottom). 

Station  Set-up  and  Pre-Collection  Checklist 


In  order  to  improve  data  collections  and  ensure  each  source  of  data  (sensors,  cameras,  etc.)  is  working  correctly 
prior  to  participant  collection,  a  pre-collection  checklist  was  created  for  each  station  (Figure  X).  It  is  currently 
still  under  review  by  the  research  team,  and  includes  but  is  not  limited  to:  working  sensors,  close-up  cameras, 
scene  cameras,  and  presence  of  narratives. 


Pre-Collection  Pre-Collection 

Checklist  Checklist 


Station  A 


I  I  Confirmed  all  sensors  work 

Close-up  camera  displayed  in  Motion 
fl  Monitor  screen  (to  the  right  of  the 
scertecam) 

□  Scene  camera  adjusted  to  view  the 
whole  simulation 

□  Ensure  narratives  and  cognitive 

scenarios  are  all  present  and  in  order 

□  Administrator  approved  final 
simulation  setup 


Station  B 


I  I  Confirmed  all  sensors  work 
Close-up  camera  displayed  in 
I  I  Motion  Monitor  screen  (to  the 
right  of  the  scene  cam) 

□  Scene  camera  adjusted  to  view 
the  whole  simulation 

□  Injuries  made  and  pre-pictures 
taken 

I  I  Narrative  and  scenario  present 

□  Administrator  approved  final 
simulation  setup 


Pre-Collection 

Checklist 


Station  C 


I  I  Confirmed  all  sensors  work 

_  Close-up  camera  displayed  in  Motion 

I  I  Monitor  screen  (to  the  right  of  the 
sceite  cam) 

□  Scene  camera  adjusted  to  view  the 
whole  simulation 

□  Ensure  narratives  and  cognitive 

scenarios  are  all  present  and  in  order 

□  Administrator  approved  final 
simulation  setup 


Pre-Collection 

Checklist 

Station  D 


r~1  Confirmed  all  sensors  work 
Internal  camera  displayed  in 
|~~|  Motion  Monitor  screen  (to  the 
right  of  the  scene  cam) 

□  Scene  camera  adjusted  to  view 
the  whole  simulation 

□  Internal camera  and  kght  source 
tested  on  monitor 

I  I  Narrative  and  scenario  present 

□  Administrator  approved  final 
simulation  setup 


Figure  21:  Pre-Collection  checklist  for  Stations  A  and  B  (Left)  and  Stations  C  and  D  (Right) 
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Point  System 


In  order  to  explore  skills  decay  through  multiple  avenues,  a  point  system  is  in  development  by  use  Cognitive 
Task  Analysis.  Essential  portions  of  clinical  procedures  were  placed  on  a  Likert  scale  and  board  certified 
general  surgeons  of  various  specialty  were  asked  to  rate  the  importance  of  each  step  on  a  survey.  Once  data 
collection  is  complete,  we  will  validate  the  scores  statistically  and  work  to  assign  points  for  performance  of 
each  procedure.  Participants  will  then  be  assigned  a  point  value  based  on  their  performance  and  scores  will  be 
compared. 

Smiill  Bowel  Reptiir 

Importance  of  Indhidual  Procedural  Steps 

Directions: 

•  Rate  each  step'slex'el  ofimpoftanceto  the  oxTiall  success  of  the  procedure 

•  Indicate  in  the  shaded  column  if  the  step  decision  is  absolutely  critical  to  the  success  of  therepair 
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If  you  were  to  use  a  permanent  suture  in  a  bowel  repair,  what  type  of  permanent  suture  would  you  use?  Are  there 
any  permanent  sutures  that  are  unacceptable? 


In  your  assessment  of  the  boweL  if  there  niU  be  greater  than  50*/o  luminal  narroicing.  which  of  the  foUowing  repar 
methods  w  ould  you  consider  acceptable?  fcAoose  off  rhora/rp?r> 

A.  c  Resection  and  Reanastomosis 

B  c  Primary  Closure  of  Each  faijury  Individu^y 

C.  □  Resection  ofUssue  between  Injuries  and  Primary  Closure 

D.  c  Incision  ofTissue  between  Injuries  and  Primary  Closure 

E.  o  Other  (plaosespecf^) 


Figure  22:  Example  of  survey  given  to  general  surgeons  to  develop  scoring  system 

Minor  changes  to  Retired  Surgeons’  Workbook 

The  retired  surgeons’  workbook  was  modified  to  include  questions  regarding  the  number  of  years  since  each 
retired  surgeon  has  performed  each  procedure  (Figure  X).  This  allows  for  a  clearer  understanding  of  the 
experience  level  each  retired  surgeon  has  and  how  long  it  has  been  since  they  have  had  any  exposure  to  that 
procedure  type. 


Please  estimate  the  number  o(  times 
observed 


1.  Subclavian  central  line  placement 


2.  Small  bowrel  repair 


3.  Urinary  catheter  insertion 


Please  estimate  the  number  ot  times  that  you  have  performed  and 
observed  the  followlnf  procedures,  as  well  as  the  number  of  years  since 
you  last  performed  the  procedure: 


I _ I 

Subfect  10 


.  Subclavian  central  line  placement 


3.  Urinary  catheter  in: 


Figure  23:  Modification  to  Retired  Surgeons '  Participant  Workbooks.  An  additional  column  was  added  to  the  original  workbook 
(Left)  to  include  the  number  of  years  since  each  procedure  was  performed  (Right) 
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Retired  Surgeon  Post-Collection  Survey 

Retired  surgeon  participants  will  receive  a  post-collection  survey  approximately  one  week  after  their 
participation  to  understand  what  surgical  knowledge  came  back  to  them  in  the  days  following  the  data 
collection.  This  document  will  be  loosely  modeled  after  the  Retired  Surgeon  Workbook.  We  are  currently 
developing  what  the  survey  should  look  like  and  what  areas  it  should  address.  We  will  be  submitting  the 
document  for  IRB  approval  shortly  after  the  New  Year. 

Retired  Surgeon  Post-Collection  Survey 

Retired  surgeon  participants  will  receive  a  post-collection  phone  survey  approximately  five  days  after  their 
participation  to  understand  what  surgical  knowledge  came  back  to  them  in  the  days  following  the  data 
collection.  This  document  was  originally  modeled  after  the  Retired  surgeon  workbook.  After  much  discussion 
we  realized  it  did  not  achieve  the  intended  goal  of  investigating  the  clinical  knowledge  that  returned  to  a 
surgeon.  We  have  developed  a  new  survey  that  is  now  IRB  approved. 


Figure  24:  Development  of  Retired  Surgeon  Workbook  (Top)  Modeled  After  Retired  Surgeon  Workbook  (Bottom) 


Figure  25:  Shown  above  is  the  post  procedure  interview  performed  on  retired  surgeons  to  better  understand  what  knowledge 

returns  one  to  three  days  after  a  procedure 
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New  Data  Collection  Method 


For  the  central  line  station,  we  realize  the  value  of  attaining  needle  angle  used  to  access  the  subclavian  venous 
system.  Proper  needle  angle  is  essential  as  using  too  large  of  an  angle  can  lead  to  significant  patient  morbidity. 
As  such,  we  have  sensitized  the  syringe  attached  to  the  needle  and  the  simulator  so  that  we  may  accurately 
measure  this  needle  angle. 


Figure  26:  New  sensorization  of  syringe  and  simulator  to  acquire  needle  angles 


Additional  Staffing 

One  additional  staff  member  has  been  added  to  the  research  team  over  the  course  of  the  last  year.  He  brings  a 
specific  background  and  expertise  to  the  team,  which  will  overall  assist  with  addressing  the  grant’s  objectives. 

Hossein  Mohamadipanah: 

Hossein  Mohamadipanah  received  his  PhD  in  Mechanical  and  Aerospace  Engineering  from  Oklahoma  State 
University.  His  main  interests  include  machine  learning  and  medical  robotics  in  the  training  and  assessment  of 
surgical  skills.  With  his  background,  he  will  have  a  critical  role  in  the  extensive  and  high-level  analyses  of 
motion  data  collected  from  participants  in  this  study.  In  addition,  he  will  continue  cross-analyses  of  clinical 
simulation  performance  and  virtual  reality  performance. 
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Key  Research  Accomplishments 

Table  1  below  provides  a  bulleted  list  of  the  project  accomplishments  as  organized  by  the  quarter  they  were 
achieved.  Additional  information  such  as  research  citations  and  specific  achievement  dates  have  been 
provided  whenever  they  were  available. 


Table  1.  List  of  Pro  ject  Accomplishments  per  Quarter 


# 

Task 

Quarter 

1 

Continued  updating  “Station  Manager  Binders”  to  reflect  best  practices  including  more  detailed 
inventory  and  organization  to  the  mobile  units.  New  protocols  were  documented  for  how  to  set 
up  and  run  each  of  the  four  clinical  stations.  Pre-  and  post-  collection  checklists  were  also  added 
to  ensure  thoroughness  and  organization.  Binder  contents  were  specialized  and  further  detailed 
for  each  specific  station. 

One 

2 

Updated  Survey  Workbook  to  make  clear  resident  Post-Graduate  Year,  clinical  and  research 
years  completed,  as  well  as  past  procedural  experience. 

One 

3 

Updated  REDCap  Data  Base  for  2015  data  coding.  Data  variables  clarified  to  reflect  station 
changes,  new  relevant  time  points,  and  significant  common  events.  All  modules  re-organized  for 
enhanced  ease  of  data  import  and  export  by  researchers. 

One 

4 

Updated  LVH  Pretest  Instructions  to  gather  more  informative  and  clear  information  on  hernia 
port  placement.  The  pretest  now  differentiates  between  ports  for  instruments  and  ports  for 
cameras. 

It  also  asks  participants  to  assign  the  order  in  which  they  would  place  the  ports  used. 

One 

5 

Hired  4  new  undergraduate  students  to  assist  in  2015  data  collection  and  data  coding. 

One 

6 

Continued  training  and  cross  training  new  personnel  into  Clinical  Station  Teams  as  the  needs  for 
scheduling  the  data  collections  changed  based  on  location  and  dates.  These  teams  were 
organized  to  include  station  leaders,  experienced  and  novice  researchers. 

One 

7 

Continued  to  Train  Staff  on  Using  Audio/Video  transcription  software  Transana.  Software  has 
been  used  for  coding  and  categorizing  quantitative  and  qualitative  features  from  verbal  and 
video  recorded  data.  Three  undergraduate  students  have  been  fully  trained  on  using  software 
and  have  completed  transcribing  audio  data  from  three  of  the  four  clinical  stations  from  2014. 

One 

8 

Submitted  IRB  update  for  approval  to  add  retired  and  practicing  surgeons  as  well  as  medical 
students. 

One 

9 

Continued  ongoing  data  analysis  for  further  conference  abstract  and  manuscript  submissions. 

One 

10 

Conducted  pilot  on  April  30,  2015.  Goals  of  this  pilot  were  to  familiarize  new  personnel  to 
stations,  test  station  enhancements  and  test  new  suture  task  which  was  incorporated  into  Station 

A. 

One 

11 

Began  second  year  data  collection  on  May  26,  2015.  Collected  from  7  residents  from  UW 
Madison;  all  of  which  were  returning  participants. 

One 

12 

Recruitment  efforts  for  summer  2015  include  capturing  as  many  returning  residents  as  possible 
as  well  as  recruiting  new,  incoming  research  residents. 

One 

13 

Updates  to  Common  Events  Checklist  for  Station  B  to  account  for  the  different  types  of  repairs 
possible.  Researchers  are  directed  to  a  list  of  questions  dependent  on  whether  the  participant 
performed  a  bowel  anastomosis  or  a  primary  enterorrhaphy  to  repair  the  injured  portion  of  small 
bowel. 

One 

14 

Collected  Data  from  46  General  Surgery  Residents  from  7  different  residency  programs  across 
the  Midwest.  16  of  these  participants  completed  the  data  collection  for  the  2"*^  year  in  a  row, 
while  30  of  these  participants  were  new  to  the  study. 

Two 
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Table  1.  List  of  Project  Accomplishments  per  Quarter 


# 

Task 

Quarter 

15 

Added  2  New  Programs  to  Our  Study.  Washington  University  in  St.  Louis,  MO  and  the  Medical 
College  of  Wisconsin  (MCW)  both  participated  for  the  first  time  in  our  study  and  expressed 
great  interest  in  continuing  participation  in  the  study  next  year. 

Two 

16 

Began  Planning  for  Retired  Surgeons  Data  Collection  beginning  this  November  at  the 

Wisconsin  Surgical  Society  (WSS)  conference.  We  have  created  marketing  packets  to  recruit 
participants,  booked  a  venue  and  began  planning  for  these  participants. 

Two 

17 

Hired  and  Trained  1  New  Undergraduate  Worker  to  assist  in  the  transcription  of  the  audio  and 
video  data  collected  from  2015  data  collections. 

Two 

18 

Hired  and  Trained  1  New  Researcher  who  assisted  in  the  summer  data  collections  and  helped 
run  our  simulation  stations. 

Two 

19 

Hired  a  New  Post  Doctorate  Researcher  to  take  leadership  positions  in  the  upper  level 
organization  and  analysis  of  the  data  collected  from  2014  and  2015  data  collections.  Hossein 
Mohamadipanah  is  in  the  process  of  being  fully  trained  on  the  central  line  catheterization 
station,  the  LVH  repair  station,  and  the  virtual  reality  station 

Two 

20 

Engaged  in  a  New  Collaboration  with  Parmesh  Ramanathan,  an  ECE  professor  in  signal 
processing  from  the  Department  of  Computer  Sciences.  Him  and  his  student  will  be  assisting  us 
in  organizing  and  analyzing  the  vast  amount  of  motion  data  collected  from  our  small  bowel 
repair  station. 

Two 

21 

Updated  Transana  Transcription  Software  with  new  and  improved  features  to  assist  our 
undergraduate  students  in  the  transcription  and  coding  of  2015  audio  and  video  data. 

Two 

Redefined  the  Clinical  Station  Teams  for  the  analysis  of  station  data.  These  teams  incorporated 

22 

experienced  members  of  the  lab  with  new  members  of  the  lab.  These  teams  will  meet 
individually  once  a  month  to  discuss  progress  made  and  future  plans  for  analyses. 

Two 

Redefined  the  Virtual  Reality  Station  Teams  for  the  analysis  of  station  E  and  F  data.  These 

23 

teams,  like  the  clinical  station  teams,  incorporate  experienced  and  new  members  of  the  lab. 

They  will  also  meet  once  a  month  to  discuss  progress  made  and  future  plans  for  analyses. 

Two 

24 

Submitted  and  Received  Approval  on  an  IRB  change  of  protocol  to  add  retired  and  practicing 
surgeons  as  well  as  medical  students  to  our  study.  This  includes  the  approval  of  our  new  survey 
materials  that  are  directed  towards  retired  surgeons  and  medical  students. 

Two 

25 

Continued  ongoing  data  analysis  for  further  conference  abstract  and  manuscript  submissions. 

Two 

Collected  Expert  Data  from  the  Urinary  Catheterization  station  as  well  as  the  Central  Line 

26 

Catheterization  station.  This  expert  feedback  and  data  will  help  us  compare  pre-mastery 
performance  data  collected  from  residents  to  post-mastery  performance  data.  It  will  also  help  us 
further  refine  stations  to  be  as  realistic  as  possible  for  retired  surgeon  data  collections  and  for 
future  resident  data  collections. 

Two 

27 

Recruited  Experts  to  Participate  in  the  Small  Bowel  Repair  station  as  well  as  the  LVH  Repair 
station.  These  collections  will  take  place  in  September  and  October  of  2015.  We  will  continue  to 
recruit  experts  from  UW  Madison  to  perform  our  stations. 

Two 

28 

Collected  Data  from  4  Retired  General  Surgeons  here  at  UW  Madison.  Data  from  retired 
surgeons  will  be  used  as  a  comparison  of  pre-  vs.  post-mastery  skills  decay. 

Three 

29 

Created  a  Retired  Surgeons’  RedCAP  database  for  the  coding  of  all  retired  surgeon  data- 
separate  from  the  database  where  resident  data  is  coded. 

Three 

30 

Recruited  Retired  Participants  from  Wisconsin  Surgical  Society  (WSS)  Conference  in  Kohler, 

WI.  Succeeded  in  getting  the  names  and  email  addresses  of  7  potential  retired  general  surgeons 
who  are  interested  in  participating  in  the  study. 

Three 

31 

Continued  Recruiting  for  Retired  Surgeons  Data  Collection  with  individuals  interested  from  the 
WSS  Conference,  as  well  as  referrals  from  past  participants. 

Three 
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Table  1.  List  of  Project  Accomplishments  per  Quarter 


# 

Task 

Quarter 

32 

Hired  and  Trained  1  New  Undergraduate  Worker  to  assist  in  the  transcription  of  the  audio  and 
video  data  collected  from  2015  data  collections  as  well  as  the  coding  of  2015  Urinary 
Catheterization  data. 

Three 

Discussed  future  plans  and  analyses  related  to  this  study  at  an  all-day  “DoD  Retreat”.  Data  was 

33 

presented  to  the  group  in  regards  to  our  resident  demographics,  clinical  stations,  as  well  as 
virtual  reality  stations.  Collaborators  Felix  Huang  and  Sandro  Mussa-Ivaldi  from  Northwestern 
joined  in  the  discussion  with  our  lab  in  planning  future  analyses  and  the  overall  future  of  the 
study. 

Three 

34 

Trained  New  Undergraduate  Students  on  Transana  software  to  assist  in  the  transcription  and 
coding  of  2015  audio  and  video  data 

Three 

35 

Continued  ongoing  data  analysis  for  further  conference  abstract  and  manuscript  submissions. 

Three 

36 

Collected  Expert  Data  from  the  Laparoscopic  Ventral  Hernia  Repair  station.  This  expert 
feedback  and  data  will  help  us  compare  pre-mastery  performance  data  collected  from  residents 
to  post-mastery  performance  data.  It  will  also  help  us  further  refine  stations  to  be  as  realistic  as 
possible  for  retired  surgeon  data  collections  and  for  future  resident  data  collections. 

Three 

37 

Recruited  Expert  to  Participate  in  the  Small  Bowel  Repair  station.  This  collection  will  take  place 
in  January  of  2016.  We  will  continue  to  recruit  experts  from  UW  Madison  to  perform  our 
stations. 

Three 

38 

Recruited  Experts  to  Validate  scoring  algorithms  for  performance  on  the  Subclavian  Central 

Line  Insertion,  Small  Bowel  Repair,  and  Urinary  Catheter  Insertion.  Validation  of  these  scoring 
algorithms  will  continue  into  Spring  of  2016. 

Three 

39 

Collected  Data  from  4  Retired  General  Surgeons  here  at  UW  Madison.  Data  from  retired 
surgeons  will  be  used  as  a  comparison  of  pre-  vs.  post-mastery  skills  decay. 

Four 

40 

Created  a  2016  RedCAP  database  for  the  coding  of  all  2016  resident  data. 

Four 

41 

Recruited  Retired  Participants  from  the  Wisconsin  Medical  Society  and  the  American  College 
of  Surgeons.  Succeeded  in  getting  the  names  and  addresses  of  over  200  potential  retired  general 
surgeons  in  Wisconsin  and  Illinois. 

Four 

42 

Hired  and  Trained  3  New  Undergraduate  Workers  to  assist  in  the  collection  and  coding  of 
retired  surgeon  data,  as  well  as  2016  resident  data. 

Four 

43 

Trained  Undergraduate  Students  on  running  clinical  simulation  stations  as  well  as  motion 
monitor  software. 

Four 

44 

Continued  ongoing  data  analysis  for  further  conference  abstract  and  manuscript  submissions. 

Four 

45 

Recruited  Experts  to  Participate  in  the  full  data  collection  this  upcoming  summer. 

Four 

46 

Recruited  Experts  to  Validate  scoring  algorithms  for  performance  on  the  Subclavian  Central 

Line  Insertion,  Small  Bowel  Repair,  Urinary  Catheter  Insertion,  and  Laparoscopic  Ventral 

Hernia  Repair.  Analyses  of  these  scoring  algorithms  will  continue  into  Summer  of  2016. 

Four 

47 

Presented:  4  abstracts  to  the  American  Surgical  Congress  in  2016 

Four 
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Reportable  Outcomes 

Table  2  provides  a  bulleted  summary  of  the  reportable  outcomes  achieved  over  the  past  year.  These 
outcomes  include  specific  methodological  improvements,  new  prototypes  such  as  the...,  and  also  specific 
products  such  as  conference  presentations,  research  papers,  assessment  instruments,  and  research  protocols. 
Specific  citations  have  been  provided  when  appropriate. 


Table  2.  List  of  Reportable  Outcomes  per  Quarter  (Type:  Product  or  Methodology) 


# 


Type 


Outcome 


Quarter 


1  Methodology 


Product 


One 


One 


Station  A:  Subclavian  Central  Line.  The  general  setup  for  this  station  remained 
the  same;  however,  a  new  task  was  added  to  the  time  rotation  for  this  station’s 
effort.  An  additional  camera  was  affixed  to  a  post  mounted  to  this  station’s  table 
to  capture  a  close-up  view  of  the  participant’s  placement  of  needles  on  the 
simulation  model.  This  assists  greatly  in  the  post-processing  video  analysis  of 
correct  needle  placement. 

Suturing  Task.  An  additional  task  was  added  to  the  clinical  procedural 
scenarios.  This  task  is  an  isolated  and  constrained  task  of  placing  3  interrupted 
sutures  into  body  tissue  mimics  using  an  instrument  tying  technique.  This  work 
is  based  off  of  a  task  developed  and  refined  in  Anne-Lise  D’Angelo’s  NIH  F32 
training  grant  project  using  the  Variable  Tissue  Simulator.  The  task  simulates 
differing  levels  of  task  complexity  by  presenting  two  different  materials  to 
suture:  rubber  balloon  (dense  connective  tissue),  tissue  paper  (friable  tissue). 
The  addition  of  this  station  is  highly  useful  in  establishing  a  baseline  of  motor 
performance  in  suturing  while  in  an  isolated  contextual  environment.  This  can 
then  be  used  as  a  comparison  to  identify  differences  in  performance  when 
suturing  is  investigated  in  the  performance  of  the  Bowel  Anastomosis  station. 
Another  benefit  offered  by  this  station  comes  from  the  data  analysis  methods 
Product  refined  for  D’Angelo’s  experimental  results.  The  suture  board  presented  here  is 

identical  to  that  of  her  project,  so  the  analysis  of  the  motion  capture  data 
matches  very  well.  The  analysis  programs  written  for  her  project  can  be  directly 
applied  to  data  collected  with  this  additional  station  task. 

The  performance  of  this  Suturing  Task  was  added  to  Station  A  because  the 
Central  Line  placement  task  performance  is  typically  fast  for  a  participant.  The 
spare  time  compared  to  other  stations  was  a  good  place  to  add  a  second  task, 
since  participants  are  already  wired  into  the  motion  capture  system.  The 
Suturing  Task  fixture  was  fabricated  to  be  placed  directly  on  top  of  the  Central 
Line  fixture,  facilitating  rapid  change-overs.  The  workspace  provides 
participants  with  a  suture  board  containing  materials  and  trays  that  hold 
necessary  instruments.  Additionally,  a  camera  mounted  to  a  support  post 
captures  a  close-up  view  of  the  participant’s  suturing  actions. 

Station  B:  Bowel  Anastomosis.  An  additional  camera  was  placed  into  this 
station’s  workspace  to  capture  close-up  views  of  the  participant’s  hand 
movements  as  they  perform  the  anastomosis  task.  This  helps  to  provide  greater 
Methodology  detail  of  the  bowel  exploration  and  specific  suturing  methods  participants 
choose  to  use  during  the  task  execution.  The  camera  is  affixed  to  a  post 
mounted  to  the  station’s  table.  This  is  located  near  the  right  shoulder  of  the 
experimenter  assistant  that  interacts  with  the  subject. 

Station  C:  Urinary  Catheterization.  Based  on  feedback  obtained  from  focus 
group  interviews  with  past  participants,  this  station  underwent  a  major  redesign 


One 


One 
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in  model  positioning  and  equipment  placement.  First,  the  orientation  of  the 
pelvic  model  was  placed  on  the  table  in  a  fashion  to  mimic  the  positioning  of  a 
patient  lying  on  a  bed.  This  better  simulated  the  leaned  over  position  the 
participant  would  have  to  do  if  in  a  patient’s  room.  Second,  a  new  catheter  kit 
was  made  to  better  match  the  way  a  clinician  would  place  a  Foley  kit  between  a 
patient’s  legs  prior  to  starting  a  urinary  catheterization.  The  kit  approximates 
the  purchased  kit  and  also  facilitates  rapid  changeovers  because  it  can  be  picked 
up  and  switched  out  with  matching  kits  that  contain  different  size  catheters  that 
can  be  requested.  Additionally,  a  close-up  camera  and  mount  was  added  to  this 
station  to  better  capture  the  detail  in  the  participant’s  catheter  insertion. 
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Addition  of  Suture  Task  In  order  to  further  examine  the  components  of  some  of 
the  more  complicated  manual  procedures  such  as  the  bowel  anastomosis,  we 
have  added  a  short  suture  task  in  which  participants  will  suture  together  two 
pieces  of  material  that  simulate  frail  tissue  and  firmer  skin  tissue.  The  hand 
technique,  efficiency  and  pauses  will  be  analyzed. 

Addition  of  Close  up  Cameras  to  three  of  the  four  clinical  stations.  These  allow 
for  a  more  detailed,  focused  view  of  the  scene  and  assist  in  coding  errors  and 
events.  They  each  have  a  standardized  position/view  and  are  manually  focused 
so  that  participant  movement  does  not  blur  the  view  of  the  simulation. 

Updates  to  the  Urinary  Catheterization  Station 

a.  Simulation  models  turned  sideways  (no  longer  facing  edge  of  table)  to 
mimic  position  of  actual  patient  awaiting  catheter  insertion. 

b.  Two  models  used  for  catheterization  instead  of  three  due  to  time 
constraints. 

c.  Vaginal  ring  added  to  female  simulator  to  restrict  access  to  urethra; 
previously  sutures  were  used  to  restric  access,  however  these  were 
unrealistic  and  prone  to  breaking. 

d.  Yellow  and  red  colored  water  instead  of  clear  water  used  to  represent 
normal  and  bloody  urine  return,  respectively. 

e.  Drainage  bags  removed  from  ports  to  increase  participant  turn-over  and 
facilitate  station  clean-up. 

f.  Catheter  boxes/kits  created  to  replace  catheter  stand. 

Modified  Protocol  to  Standardize  POV  Cameras  Video  glasses  with  low  image 
quality  were  retired  and  replaced  by  additional  head-mounted  cameras.  These 
cameras  were  set  up  to  have  a  better  focus  and  better  image  quality  than 
previously  recorded  point-of-view  video  data. 


One 


One 


One 


One 


Documentation  of  Technological  Protocols 

With  the  addition  of  new  equipment  such  as  the  close-up  cameras,  and 
standardization  of  protocols  with  head  cameras  and  camera  mounting,  a  number 
of  new  training  documents  were  created.  These  cover  in  detail  the  physical 

9  Product  setup  of  technology  devices,  placement  in  the  operating  workspace,  software  One 

configurations,  and  use  during  data  collections.  Each  of  the  station  personnel 
responsible  for  setup  has  been  trained  in  equipment  using  these  instructions. 

Highlighted  below  are  portions  of  documents  concerning  head  and  close-up 
camera  setup. 

Station  C:  Urinary  Catheterization.  One  of  the  scenarios  at  this  station  involves  a 
patient  with  Benign  Prostatic  Hypertrophy  (BPH).  Because  current  market 

10  Product  bladder  catheterization  models  do  not  account  for  this  variation  in  patient  Two 

anatomy,  we  are  currently  developing  a  model  that  improves  on  our  current  BPH 
one  that  would  evaluate  catheter  insertion  technique  and  procedural  decision 
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making.  The  current  BPH  model  was  originally  developed  as  a  modification  to 
an  available  market  bladder.  We  are  interested  in  improving  the  realism  of  the 
model,  so  that  the  struggles  healthcare  providers  face  when  placing  a  catheter  in 
a  patient  with  BPH  are  more  accurate  in  feel  and  location  when  using  our  BPH 
model.  Recently,  8  urology  residents  tested  out  the  model  to  see  if  the  model 
replicated  a  real  experience  with  a  patient  experiencing  BPH.  We  are  hoping  that 
this  model  replaces  the  current  one  during  2016  data  collection. 

Station  Narratives  Station  A:  Central  Line  Placement.  The  introduction  narrative 
that  is  read  aloud  by  researchers  at  Station  A  was  shortened.  This  was  done 
because  participants  were  getting  confused  from  all  of  the  information  presented 
to  them  in  the  beginning  of  the  station.  Information  that  was  not  considered 
pertinent  was  eliminated  from  the  narrative  in  order  to  keep  residents  present  in 
the  situation. 

12  Methodology 

Station  B:  Bowel  Anastomosis.  Originally  the  station  involved  residents  using 

clips  to  hold  the  bowel  as  they  performed  resections  and  anastomoses.  For  the 

purposes  of  understanding  resident  performance  more  through  guiding  and 

instructing  a  researcher,  an  assistant  researcher  was  placed  at  the  Bowel  Two 

Anastamosis  station.  By  introducing  the  role  of  an  assistant  to  the  resident  at  this 

station,  the  resident  could  interact  with  them  and  provide  more  of  their 

knowledge  verbally  through  direction. 

Station  C:  Urinary  Catheterization.  In  order  to  better  assess  resident  decision 
making,  this  station  underwent  a  redesign  in  equipment  placement.  In  2014, 

13  Methodology 

residents  were  able  to  see  all  catheter  options  available  to  them.  This  year 
residents  were  required  to  vocalize  their  catheter  choice  without  visualization  of 
the  catheters  ahead  of  time.  Only  if  residents  were  unable  to  articulate  their 
decision  did  we  provide  a  list  of  catheter  options. 

Documentation  Station  B:  Bowel  Anastomosis.  In  order  to  improve  post¬ 
performance  analysis  of  bowel  repairs,  the  station’s  common  events  checklist 

14  Product 

was  revamped  to  consider  the  different  choices  residents  made,  including 
planned  layers,  suture  type,  stitch  type,  completeness,  and  general  procedure 
steps.  This  level  of  detail  will  also  expedite  statistical  analyses  performed  within 
the  station  and  across  stations. 

15  Product 

Station  C.  Urinary  Catheterization.  Based  on  experiences  researchers  have  had 
with  residents  in  previous  data  collections  on  Station  C,  a  Frequently  Asked 

Questions  document  was  created  to  include  questions  that  participants  frequently 
ask  about  the  models  and  urine  output.  By  making  this  document,  researchers 
would  have  standard  response  options  when  prompted  with  these  frequent 
questions. 

Resident  Participant  Workbook  Addition.  The  resident  workbook  was 
supplemented  this  year  with  questions  regarding  the  number  of  procedures 
performed  and  observed  by  each  resident.  This  allows  for  a  clearer 

16  Product 

understanding  of  the  experience  level  each  resident  has  and  what  previous  Two 

experience  they  have  with  each  station.  Additionally,  the  residents  were  asked  to 
rate  their  confidence  in  different  suturing  techniques  and  with  different  tissue 

17  Product 

types. 

Minor  Changes  to  Resident  Participant  Workbooks. 

•  Resident  participant  workbook  pre-surveys  were  reorganized  and  reworded  so 

that  questions  were  clearer  and  also  to  clarify  follow-up  questions.  Two 

•  Resident  participant  final  exit  surveys  were  modified  to  include  questions 
regarding  whether  or  not  they  would  use  training  materials  while  they  were 
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away  from  clinical  rotation  to  reduce  skills  decay. 

Retired  Surgeon  Workbooks.  The  retired  surgeon  workbooks  were  developed 

based  on  the  resident  participant  workbooks;  however,  the  survey  questions 

were  modified  to  match  a  retired  surgeon’s  experience  and  lifestyle.  The  goal 

was  to  keep  as  much  information  in  the  retired  surgeon  workbooks  the  same  as  Two 

the  resident  participant  ones  so  that  analyses  could  be  performed  across  groups. 

The  workbook  and  its  modifications  were  submitted  and  approved  by  the 

University  of  Wisconsin  Institutional  Review  Board. 

Station  A:  Central  Line  Catheterization.  In  a  retired  surgeon  data  collection,  we 
tried  adding  an  extra  camera  to  the  overhead  frame  for  additional  methods  of 

19  Product 

analysis.  This  additional  camera  was  not  linked  to  motion  monitor  but  provided  xhree 

a  better  quality  overhead  view  of  the  participant  and  surrounding  scene.  The 
recordings  from  this  camera  were  used  to  test  a  new  data  analysis  methodology 
for  instrument  tracking. 

Lucas-Kanade  Template  Matching  Tracking  Algorithm.  Because  we  have  a 
significant  amount  of  data  in  video  format,  we  have  started  using  the  Lucas- 
Kanade  algorithm  to  automate  the  identification  of  instruments.  It  is  based  on 

20  Methodology 

template  matching  between  two  consecutive  frames  and  uses  multiple  Three 

parameters  to  identify  transformation  of  feature  frames.  We  were  able  to  test  this 
algorithm  on  Station  A  with  the  scalpel  used  to  perform  the  central  line 
catheterization  (Figure  X). 

Station  Set-up  and  Pre-Collection  Checklist.  In  order  to  improve  data  collections 
and  ensure  each  source  of  data  (sensors,  cameras,  etc.)  is  working  correctly  prior 

21  Methodology 

to  participant  collection,  a  pre-collection  checklist  was  created  for  each  station  Three 

(Figure  X).  It  is  currently  still  under  review  by  the  research  team,  and  includes 
but  is  not  limited  to:  working  sensors,  close-up  cameras,  scene  cameras,  and 
presence  of  narratives. 

22  Product 

The  retired  surgeons’  workbook  was  modified  to  include  questions  regarding  the 
number  of  years  since  each  retired  surgeon  has  performed  each  procedure.  This 
allows  for  a  clearer  understanding  of  the  experience  level  each  retired  surgeon  Three 

has  and  how  long  it  has  been  since  they  have  had  any  exposure  to  that  procedure 

23  Methodology 

type. 

Retired  surgeon  participants  will  receive  a  post-collection  survey  approximately 

one  week  after  their  participation  to  understand  what  surgical  knowledge  came 

back  to  them  in  the  days  following  the  data  collection.  This  document  will  be  Three 

loosely  modeled  after  the  Retired  Surgeon  Workbook.  We  are  currently 

developing  what  the  survey  should  look  like  and  what  areas  it  should  address. 

We  will  be  submitting  the  document  for  IRB  approval  shortly  after  the  new  year. 

24  Methodology 

Point  system:  Clinical  Stations.  In  order  to  explore  skills  decay  through  multiple 
avenues,  a  point  system  is  in  development  by  use  Cognitive  Task  Analysis. 

Essential  portions  of  clinical  procedures  were  placed  on  a  Likert  scale  and  board 
certified  general  surgeons  of  various  specialty  were  asked  to  rate  the  importance 
of  each  step  on  a  survey.  Once  data  collection  is  complete,  we  will  validate  the 
scores  statistically  and  work  to  assign  points  for  performance  of  each  procedure. 

Participants  will  then  be  assigned  a  point  value  based  on  their  performance  and 
scores  will  be  compared. 

Retired  Surgeon  Post-Collection  Survey.  Retired  surgeon  participants  will 

25  Methodology 

receive  a  post-collection  phone  survey  approximately  seven  days  after  their  p^^^ 

participation  to  understand  what  surgical  knowledge  came  back  to  them  in  the 
days  following  the  data  collection.  This  document  was  originally  modeled  after 
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the  Retired  surgeon  workbook.  After  much  discussion  we  realized  it  did  not 
achieve  the  intended  goal  of  investigating  the  clinical  knowledge  that  returned  to 
a  surgeon.  We  have  developed  a  new  survey  that  is  now  IRB  approved. 


New  Data  Collection  Method  For  the  central  line  station,  we  realize  the  value  of 
attaining  needle  angle  used  to  access  the  subclavian  venous  system.  Proper 
26  Methodolo  n^^dle  angle  is  essential  as  using  too  large  of  an  angle  can  lead  to  significant 

patient  morbidity.  As  such,  we  have  sensitized  the  syringe  attached  to  the  needle 
and  the  simulator  so  that  we  may  accurately  measure  this  needle  angle. 


Final  Conclusions 


Year  three  of  the  project  has  included  a  number  of  significant  steps  towards  meeting  the  four  key  study 
objectives  outlined  by  our  original  SoW.  Specifically,  the  team  has  successfully  completed  the  second  year  of 
data  collection;  refined  the  simulation  stations  for  the  initial  and  future  data  collections;  analyzed  performance 
on  multiple  stations;  and  disseminated  our  preliminary  work  in  the  form  of  several  papers  and  presentations.  We 
have  moved  towards  collecting  data  from  retired  surgeons  and  completing  our  final  year  of  data  collections  with 
research  residents.  Using  the  no  cost  extension  to  finalize  our  data  analysis  and  move  towards  completing  work 
on  all  four  areas  of  our  SoW  will  be  imperative  to  the  final  portion  of  our  research. 
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